Trans-dimensional Random Fields for Language Modeling
نویسندگان
چکیده
Language modeling (LM) involves determining the joint probability of words in a sentence. The conditional approach is dominant, representing the joint probability in terms of conditionals. Examples include n-gram LMs and neural network LMs. An alternative approach, called the random field (RF) approach, is used in whole-sentence maximum entropy (WSME) LMs. Although the RF approach has potential benefits, the empirical results of previous WSME models are not satisfactory. In this paper, we revisit the RF approach for language modeling, with a number of innovations. We propose a trans-dimensional RF (TDRF) model and develop a training algorithm using joint stochastic approximation and trans-dimensional mixture sampling. We perform speech recognition experiments on Wall Street Journal data, and find that our TDRF models lead to performances as good as the recurrent neural network LMs but are computationally more efficient in computing sentence probability.
منابع مشابه
Discrete fast algorithms for two-dimensional linear prediction on a polar raster
New generalized split Levinson and Schur algorithms for the two-dimensional linear least squares prediction problem on a polar raster are derived. The algorithms compute the prediction filter for estimating a random field at the edge of a disk, from noisy observations inside the disk. The covariance function of the random field is assumed to have a Toeplitzplus-Hankel structure for both its rad...
متن کاملDCT/DST and Gauss-Markov fields: conditions for equivalence
The correspondence addresses the intriguing question of which random models are equivalent to the discrete cosine transform (DCT) and discrete sine transform (DST). Common knowledge states that these transforms are asymptotically equivalent to first-order Gauss causal Markov random processes. We establish that the DCT and the DST are exactly equivalent to homogeneous one-dimensional (1-D) and t...
متن کاملA likelihood ratio formula for two-dimensional random fields
This paper is concerned with the detection of a random signal in white Gaussian noise when both the signal and the noise are two-dimensional random fields. The principal result is the derivation of a recursive formula for the likelihood ratio relating it to certain conditional moments of the signal. It is also shown that, except for some relatively uninteresting cases, a simple exponential form...
متن کاملLearning neural trans-dimensional random field language models with noise-contrastive estimation
Trans-dimensional random field language models (TRF LMs) where sentences are modeled as a collection of random fields, have shown close performance with LSTM LMs in speech recognition and are computationally more efficient in inference. However, the training efficiency of neural TRF LMs is not satisfactory, which limits the scalability of TRF LMs on large training corpus. In this paper, several...
متن کاملThe evanescent field transform for estimating the parameters of homogeneous random fields with mixed spectral distributions
Parametric modeling and estimation of complex valued homogeneous random fields with mixed spectral distributions is a fundamental problem in two-dimensional (2-D) signal processing. The parametric model under consideration results from the 2-D Wold-type decomposition of the random field. The same model naturally arises as the physical model in problems of space-time adaptive processing of airbo...
متن کامل